from sklearn_benchmarks.reporting.hp_match import HpMatchReporting

scikit-learn vs. ONNX Runtime

reporting = HpMatchReporting(other_library="onnx", config="config.yml", log_scale=True)
reporting.make_report()

We assume here there is a perfect match between the hyperparameters of both librairies. For a given set of parameters and a given dataset, we compute the speedup time scikit-learn / time onnx. For instance, a speedup of 2 means that onnx is twice as fast as scikit-learn for a given set of parameters and a given dataset.

KNeighborsClassifier_brute_force

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=brute.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	2.037486	0.160038	NaN	...	brute	-1	1	0.663	0.316630	0.007363	1.000	6.434903	6.436643	0.337
2	KNeighborsClassifier_brute_force	predict	100000	1	100	e70d8e7886766d41f30889506baa1e67	2cf49829391aa7891e877f2ed070adf0	0.021217	0.001287	NaN	...	brute	-1	1	1.000	17.037362	0.208360	0.757	0.001245	0.001245	0.243
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	22d5eac4a4146149b35eae6914921514	2cf49829391aa7891e877f2ed070adf0	2.840747	0.033275	NaN	...	brute	-1	5	0.757	17.163153	0.117021	0.882	0.165514	0.165518	0.125
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	2.063801	0.004240	NaN	...	brute	1	100	0.882	0.324414	0.005377	1.000	6.361627	6.362501	0.118
8	KNeighborsClassifier_brute_force	predict	100000	1	100	16f94fc93942ff7ece860c9b0f64645f	2cf49829391aa7891e877f2ed070adf0	0.019551	0.000110	NaN	...	brute	1	100	1.000	17.362923	0.159958	0.757	0.001126	0.001126	0.243
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2d5f2b2c02d77766434d9e5033b4a76c	2cf49829391aa7891e877f2ed070adf0	2.858204	0.019720	NaN	...	brute	-1	100	0.882	17.520432	0.029419	0.663	0.163135	0.163136	0.219
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	2.064481	0.028765	NaN	...	brute	1	5	0.757	0.245264	0.004166	1.000	8.417390	8.418604	0.243
14	KNeighborsClassifier_brute_force	predict	100000	1	100	efac583623711d395ddae7a1e881ff68	2cf49829391aa7891e877f2ed070adf0	0.019926	0.000144	NaN	...	brute	1	5	1.000	3.715633	0.014216	0.922	0.005363	0.005363	0.078
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	b79d63bb7670c492de5c3befac58fe29	2cf49829391aa7891e877f2ed070adf0	1.183130	0.005814	NaN	...	brute	1	1	0.663	3.756712	0.020056	0.929	0.314938	0.314942	0.266
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	1.815722	0.028080	NaN	...	brute	-1	1	0.896	0.243265	0.003841	1.000	7.463982	7.464912	0.104
20	KNeighborsClassifier_brute_force	predict	100000	1	2	e70d8e7886766d41f30889506baa1e67	88843f54689e3271092f70126e1de585	0.008757	0.004839	NaN	...	brute	-1	1	1.000	3.712156	0.011406	0.922	0.002359	0.002359	0.078
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	22d5eac4a4146149b35eae6914921514	88843f54689e3271092f70126e1de585	2.707666	0.022798	NaN	...	brute	-1	5	0.922	3.765046	0.017118	0.896	0.719159	0.719166	0.026

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.147	0.0	-1	1	17.404	0.036	0.663	0.001	0.001	See	See
3	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.193	0.0	-1	5	0.321	0.004	1.000	0.035	0.035	See	See
6	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.139	0.0	1	100	17.075	0.033	0.882	0.001	0.001	See	See
9	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.016	0.006	5.082	0.0	-1	100	0.322	0.005	1.000	0.049	0.049	See	See
12	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.221	0.0	1	5	3.769	0.014	0.896	0.003	0.003	See	See
15	KNeighborsClassifier_brute_force	fit	100000	100000	100	0.011	0.000	7.145	0.0	1	1	0.243	0.004	1.000	0.046	0.046	See	See
18	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.376	0.0	-1	1	3.751	0.013	0.929	0.001	0.001	See	See
21	KNeighborsClassifier_brute_force	fit	100000	100000	2	0.004	0.000	0.370	0.0	-1	5	0.244	0.004	1.000	0.018	0.018	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.037	0.160	0.000	0.002	-1	1	0.317	0.007	1.000	6.435	6.437	See	See
2	KNeighborsClassifier_brute_force	predict	100000	1	100	0.021	0.001	0.000	0.021	-1	1	17.037	0.208	0.757	0.001	0.001	See	See
4	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.841	0.033	0.000	0.003	-1	5	17.163	0.117	0.882	0.166	0.166	See	See
5	KNeighborsClassifier_brute_force	predict	100000	1	100	0.023	0.001	0.000	0.023	-1	5	0.324	0.006	1.000	0.071	0.071	See	See
7	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.064	0.004	0.000	0.002	1	100	0.324	0.005	1.000	6.362	6.363	See	See
8	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	100	17.363	0.160	0.757	0.001	0.001	See	See
10	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.858	0.020	0.000	0.003	-1	100	17.520	0.029	0.663	0.163	0.163	See	See
11	KNeighborsClassifier_brute_force	predict	100000	1	100	0.024	0.004	0.000	0.024	-1	100	0.318	0.007	1.000	0.076	0.076	See	See
13	KNeighborsClassifier_brute_force	predict	100000	1000	100	2.064	0.029	0.000	0.002	1	5	0.245	0.004	1.000	8.417	8.419	See	See
14	KNeighborsClassifier_brute_force	predict	100000	1	100	0.020	0.000	0.000	0.020	1	5	3.716	0.014	0.922	0.005	0.005	See	See
16	KNeighborsClassifier_brute_force	predict	100000	1000	100	1.183	0.006	0.001	0.001	1	1	3.757	0.020	0.929	0.315	0.315	See	See
17	KNeighborsClassifier_brute_force	predict	100000	1	100	0.018	0.000	0.000	0.018	1	1	0.243	0.005	1.000	0.076	0.076	See	See
19	KNeighborsClassifier_brute_force	predict	100000	1000	2	1.816	0.028	0.000	0.002	-1	1	0.243	0.004	1.000	7.464	7.465	See	See
20	KNeighborsClassifier_brute_force	predict	100000	1	2	0.009	0.005	0.000	0.009	-1	1	3.712	0.011	0.922	0.002	0.002	See	See
22	KNeighborsClassifier_brute_force	predict	100000	1000	2	2.708	0.023	0.000	0.003	-1	5	3.765	0.017	0.896	0.719	0.719	See	See
23	KNeighborsClassifier_brute_force	predict	100000	1	2	0.006	0.002	0.000	0.006	-1	5	0.245	0.004	1.000	0.023	0.023	See	See

KNeighborsClassifier_kd_tree

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: algorithm=kd_tree.

WARNING! Mismatch between validation scores for 12 prediction (75.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	algorithm	n_jobs	n_neighbors	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.795165	0.983172	NaN	...	kd_tree	-1	1	0.929	2.664052	0.209325	1.000	0.298480	0.299400	0.071
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	fe8267e25dadb302ed498e50fabf7ef4	1eb6cf1a720a225efb91df0b529b0510	0.002597	0.000418	NaN	...	kd_tree	-1	1	1.000	108.072549	0.000000	0.946	0.000024	0.000024	0.054
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	dc74b969bf622bc24ba3bc62c980983b	1eb6cf1a720a225efb91df0b529b0510	1.050018	0.315225	NaN	...	kd_tree	-1	5	0.946	108.741033	0.000000	0.951	0.009656	0.009656	0.005
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	5.603861	0.406256	NaN	...	kd_tree	1	100	0.951	2.665895	0.155102	1.000	2.102056	2.105611	0.049
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	41accfcdcd5c50784bedf6164b63de99	1eb6cf1a720a225efb91df0b529b0510	0.003149	0.000985	NaN	...	kd_tree	1	100	1.000	104.111674	0.000000	0.946	0.000030	0.000030	0.054
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	022f7445d43bb1dbc24dc3106c03cb93	1eb6cf1a720a225efb91df0b529b0510	3.127008	0.108438	NaN	...	kd_tree	-1	100	0.951	108.824137	0.000000	0.929	0.028735	0.028735	0.022
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	1.678828	0.200153	NaN	...	kd_tree	1	5	0.946	0.004532	0.000168	1.000	370.399824	370.652988	0.054
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	84622ba45e941db642965553529e1941	1eb6cf1a720a225efb91df0b529b0510	0.001475	0.000403	NaN	...	kd_tree	1	5	1.000	0.035546	0.000614	0.911	0.041510	0.041516	0.089
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	a9455565db1d8e052a783317c99744ff	1eb6cf1a720a225efb91df0b529b0510	0.956498	0.273766	NaN	...	kd_tree	1	1	0.929	0.061317	0.002328	0.894	15.599130	15.610365	0.035
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.023352	0.010450	NaN	...	kd_tree	-1	1	0.891	0.004551	0.000100	1.000	5.131217	5.132450	0.109
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	fe8267e25dadb302ed498e50fabf7ef4	edf7cb014c2c0910e1c3aaaf8ae633f2	0.001989	0.000219	NaN	...	kd_tree	-1	1	1.000	0.035106	0.000279	0.911	0.056646	0.056648	0.089
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	dc74b969bf622bc24ba3bc62c980983b	edf7cb014c2c0910e1c3aaaf8ae633f2	0.020542	0.000941	NaN	...	kd_tree	-1	5	0.911	0.034851	0.000510	0.891	0.589424	0.589487	0.020

12 rows × 22 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.133	0.035	0.026	0.0	-1	1	109.254	0.000	0.929	0.029	0.029	See	See
3	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.764	0.040	0.021	0.0	-1	5	2.643	0.157	1.000	1.424	1.427	See	See
6	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.838	0.074	0.021	0.0	1	100	108.933	0.000	0.951	0.035	0.035	See	See
9	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.550	0.091	0.023	0.0	-1	100	2.655	0.136	1.000	1.337	1.339	See	See
12	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.568	0.076	0.022	0.0	1	5	0.041	0.009	0.891	86.659	88.840	See	See
15	KNeighborsClassifier_kd_tree	fit	1000000	1000000	10	3.754	0.043	0.021	0.0	1	1	0.005	0.000	1.000	829.796	830.012	See	See
18	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.001	0.001	0.017	0.0	-1	1	0.060	0.000	0.894	0.015	0.015	See	See
21	KNeighborsClassifier_kd_tree	fit	1000	1000	2	0.000	0.000	0.034	0.0	-1	5	0.005	0.000	1.000	0.104	0.104	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	iteration_throughput	latency	n_jobs	n_neighbors	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.795	0.983	0.000	0.001	-1	1	2.664	0.209	1.000	0.298	0.299	See	See
2	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.000	0.000	0.003	-1	1	108.073	0.000	0.946	0.000	0.000	See	See
4	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.050	0.315	0.000	0.001	-1	5	108.741	0.000	0.951	0.010	0.010	See	See
5	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	-1	5	2.653	0.147	1.000	0.001	0.001	See	See
7	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	5.604	0.406	0.000	0.006	1	100	2.666	0.155	1.000	2.102	2.106	See	See
8	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.003	0.001	0.000	0.003	1	100	104.112	0.000	0.946	0.000	0.000	See	See
10	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	3.127	0.108	0.000	0.003	-1	100	108.824	0.000	0.929	0.029	0.029	See	See
11	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.005	0.001	0.000	0.005	-1	100	2.676	0.153	1.000	0.002	0.002	See	See
13	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	1.679	0.200	0.000	0.002	1	5	0.005	0.000	1.000	370.400	370.653	See	See
14	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	5	0.036	0.001	0.911	0.042	0.042	See	See
16	KNeighborsClassifier_kd_tree	predict	1000000	1000	10	0.956	0.274	0.000	0.001	1	1	0.061	0.002	0.894	15.599	15.610	See	See
17	KNeighborsClassifier_kd_tree	predict	1000000	1	10	0.001	0.000	0.000	0.001	1	1	0.005	0.000	1.000	0.188	0.188	See	See
19	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.023	0.010	0.001	0.000	-1	1	0.005	0.000	1.000	5.131	5.132	See	See
20	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	1	0.035	0.000	0.911	0.057	0.057	See	See
22	KNeighborsClassifier_kd_tree	predict	1000	1000	2	0.021	0.001	0.001	0.000	-1	5	0.035	0.001	0.891	0.589	0.589	See	See
23	KNeighborsClassifier_kd_tree	predict	1000	1	2	0.002	0.000	0.000	0.002	-1	5	0.004	0.000	1.000	0.423	0.423	See	See

HistGradientBoostingClassifier_best

onnx (1.10.1) vs. scikit-learn (1.0.dev0)

Speedup barplots

All estimators share the following parameters: learning_rate=0.01, n_iter_no_change=10.0, max_leaf_nodes=100.0, max_bins=255.0, min_samples_leaf=100.0, max_iter=300.0.

WARNING! Mismatch between validation scores for 1 prediction (100.0%). The observed differences can be found in the diff_accuracy_scores column. The chosen difference threshold is 0.001 for accuracy_score. See details in the dataframe below.

	estimator	function	n_samples_train	n_samples	n_features	parameters_digest	dataset_digest	mean_duration_sklearn	std_duration_sklearn	n_iter	...	max_leaf_nodes	min_samples_leaf	n_iter_no_change	accuracy_score_sklearn	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	diff_accuracy_scores
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	25b9f14bed7dd5d830c6ccd4dfebbf0c	8c8fffa8ec4b2d8de833421f9e32beab	0.114318	0.001786	300	...	100	100	10	0.824	0.375916	0.009468	1.0	0.304104	0.304201	0.176

1 rows × 25 columns

Raw results

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
0	HistGradientBoostingClassifier_best	fit	100000	100000	100	85.574	0.0	300	0.001	0.001	0.454	0.021	0.824	188.313	188.512	See	See

	estimator	function	n_samples_train	n_samples	n_features	mean_duration_sklearn	std_duration_sklearn	n_iter	iteration_throughput	latency	mean_duration_onnx	std_duration_onnx	accuracy_score_onnx	speedup	std_speedup	sklearn_profiling	onnx_profiling
1	HistGradientBoostingClassifier_best	predict	100000	1000	100	0.114	0.002	300	0.007	0.0	0.376	0.009	1.0	0.304	0.304	See	See